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SUBJECT: Challenge Procedures 

NOTE 


25X1 


25X1 


Rather than comment on 


paper of 19 November 


] 


idea of an NIO-size office of 


25X1 


on the same subject, I have written a new paper which deliberately 
avoids a rehearsal of the various arguments against challenge pro- 
cedures. I take as a point of departure the DCI's instruction to the 
ICS of Dece mber 19 73 to develop such ar procedures, (The concept, 
contrary to assertion, was not first advanced by the D/DCI/IC 

on 2 January 1974--the D/DCI/IC at that time was responding to the 
DC1, who himself was responding in part to our Middle East Post 
Mortem. ) I have also deliberately avoided a discussion of various 
proposed alternatives which, in my view, are impractical or 
undesirable (e. g. , | 
professional challengers). 


Instead, I have sought here to advocate one specific approach 
to the problem --the soundest and most sensible I could come up 
with. (And it may be that others, including] [will wish to play 25X1 
the devil with my advocacy. ) In additi on to p roviding broad considera- 
tions, I have used an idea broached by | ~| in a brief paragraph (on 25X1 

the bottom of p. 18 in his memo), developing this into a general 
proposal and suggesting some specific procedures. 


I suggest that this memo be given to the DCI for his consideration 
(presumably after we have shown it to the D/DCI/NIO). If the DCI 
concurs, I would then urge that he circulate it for later discussion 
at USIB. 


25X1 

25X1 

Chie f, PRD/ IC 

Attachment 
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MEMORANDUM 

SUBJECT: Challenge Procedures - A Proposal 


Some General Considerations 

The DCI first proposed the establishment of challenge 
procedures in October 1973, when, in the aftermath of the 
failure of intelligence to warn of the impending Arab attack 
on Israel, he and others suggested that a means be found to 
introduce some form of devil's advocacy into the Community's 
analytical proceedings/'' 

Briefly, what seems to be needed is a challenge procedure 
which would, inter alia, assist production analysts to overcome 
three occupational hazards to which, according to our post-mortem 


reports, they are occasionally subject: 

--Preconceptions: the tendency to discount information 
that runs counter to long-held convictions; 


# Specifically, the DCI stated: "The IC Staff. . .will develop regular 
systems to be implemented by the NIOs to ensure that serious 
divergent points of view and conflicting elements of information 
not be submerged by managerial fiat or the mechanism of reinforcing 
consensus. . . . Such systems will also be charged with ensuring the 
establishment of means to provide the views of devils’ advocates, 
adversary procedures, and the use of gaming techniques as 
appropriate. " ("The Performance of the Intelligence Community 
Before the Arab-Israeli War of October 1973: A Preliminary Post- 
Mortem Report, " submitted by the DCI, December 1973, p. 22. } 
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- -Reinforcing consensus: the tendency for divergent 
views of individual analysts to be submerged in a sea of 
conventional collective wisdom; 

--The current intelligence syndrome: a myopic view 
of the forest because of forced focus on current intelligence 
trees. 

The notion that some way should be found to challenge conventional 
substantive judgments has by now itself become quite conventional.’ 

But perceiving the wisdom of establishing challenge procedures does 
not lead easily into an appreciation of precisely what form such 
procedures should take. It is the who , when , how , and why of 
challenge procedures which confront those who seek improvements 
in analytical performance: 

© just who in the community (and with what credentials) 
is entitled to challenge whom? 

© just when (and vis-a-vis what) should the challengers 
perform their art, i. e. , challenge? 


* Conventional enough, at any rate, elicit challenges from those 
who see only problems in a devil ' a advocate procedure. 

2 
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© precisely how should the challenger perform, i.e. , 
through what system or medium should he present his contrary 
views ? 

© and, finally, why should the challenger risk his (and 
perhaps his sponsoring agency's) reputation for sagacity for 
the sake of representing, by definition, a singular and probably 
unpopular point of view? 

Some Guiding Principles 

Clearly, some general rules of the game should be developed. 

A challenger should possess substantial substantive competence in 
the area under exploration, but he should not be so close to the subject 
that he lacks perspective and suffers from the analytical disabilities 
listed above. He should be familiar with Community processes and, 
to a degree. Community "politics. " And he should be articulate and 
persuasive. (He need not believe his own advocacy, but he should 
be devilish enough to convince others that he does. ) Finally, and 
obviously, he should have the time to do the job properly. 

When should this paragon exercise his persuasive talents in this 

strange manner? Certainly not as a matter of routine concerning 

* 

ordinary subjects. He should confine himself to matters of high 
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moment (those of long-term significance as well as of immediate 
concern) and papers of particular consequence (NIEs, Alert Memo- 
randa, etc. ). And he should deal, full-time, only with specific 
circumstances, problems, and/or papers for a stated period (weeks 
or months). A more or less permanent assignment as a devil's 
advocate would soon sap the vitality and demolish the credibility 
of even the most enthusiastic and skillful practitioner. And the 
devil probably would, over time, withdraw from the process. 

There already is, in being, Community machinery, the NIO 
system, which could accommodate --effectively, if not easily- - 
the establishment of challenge procedures. No other component 
of the Community performs so many significant substantive functions 
for the DCI and USIB; no other component is so close to the consumers; 
and no other component is so involved in the production of important 
Community assessments. And from the point of view of the advocate 
himself, no other component could so readily provide him with the 
papers, contacts, forums, and general support necessary for the 
practice of devils' advocacy. 

Indeed, in our canvass of alternatives, we could find no other 
office or mechanism which could properly support a regular devil s 

4 
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advocate process* and. we do not — for a variety of reasons favor 
the establishment of a new institution to perform this task. 

And what of the risk of serving as a champion of unpopular 
causes? Perhaps the devil's advocate — -assured that he would 
serve only a short tour* and guaranteed a respectful (if not 
sympathetic) audience — —might find the exercise of his imagination 
and the influence he brought to bear on the weighty judgments of 
the Community reward enough. And, at the conclusion of his 
sentence, he might find some solace in the drink which the NIO 
will buy for him at Sans Souci. 

And Some Specific Proposals 

Assuming, then, that. challenge procedures are to be instituted 
within an NIO- sponsored system, we would propose the following 
specific measures: 

A. ’ The DCI or the D/DCI/NIO, in consultation with 
USIB, if appropriate, should determine if a given subject and 
paper seem sufficiently important to warrant the institution of 
official challenge procedures. (A standard country paper on, say, 
Argentina would probably not so warrant; an SNIE on, say, Soviet 
reactions to specific US courses of action* alfnost certainly would. ) 


5 
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USIB principals (and perhaps other senior figures in the Community) 
should be encouraged to nominate candidates for devil’s advocate 
positions- In consultation with the D/DCI/NIO, the D/DCI/IC, 
and others, as appropriate, the NIO responsible for a given paper 
should then appoint an intelligence officer as a substantive challenger, 
responsible as a devil's advocate for the effective presentation of 
contrary substantive views. 

1. The possible appointment of a challenger should 
be discussed early on (perhaps during Terms of Reference 
meetings) with the various agency representatives and with 
USIB. 

2- The designated devil's advocate should be an 
intelligence officer who has appropriate substantive credentials, 
experience, and seniority. He could be drawn from any 
element of the Community and might, in some circumstances, 
be another NIO, perhaps one with related substantive responsibilities 

3. The devil's advocate should play his role for the 

life of the paper concerned- -in the case of an NIE, from the TR 

stage through USIB consideration; in the case of an Alert Memo- 

«* 

randum, from its inception through the DCJ^s approval. He 
should, in effect, serve a temporary tour in the devil's advocate 
capacity. 5 
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4. There are of course practical obstacles to 

« 

such a procedure. Few if any components will feel free 
to spare a senior officer for devil's advocate duty. The DCI 
may thus wish to ask US1B principals to consider this 
procedure to be a necessary part of USIB's set of substantive 
and estimative responsibilities--and, in effect, as significant 
as providing, for example, representatives to attend meetings 
on NIEs. 

5. In some instances (e.g., NIE 11-3/8), the NIO 
might wish to consider the appointment of more than one 
devil's advocate, depending on the size, complexity, and 
diversity of the paper under preparation. 

B. The challenger would be charged with: developing and 
presenting plausible arguments against the conventional wisdom 
and against any or all of the papers' major judgments; expressing 
disbelief or skepticism about certain specific pieces of evidence 
and/or discrete conclusions based on them; challenging the logic 
and coherence of given lines of argument; and identifying any gaps 
in information and "holes" in argumentation \yhich he perceived 
as affecting major judgments. . 

7 
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1. The devil’s advocate would be responsible for 
eliciting the views of other responsible officers in the 
Community who held views contrary to those offered in 
the paper under review (contrary to those offered in 
expressions of dissent as well as in the main text). 

2. The devil’s advocate should present his 
advocacy orally, during meetings on the paper, and, when 
appropriate, in writing. The NIO in charge would be duty- 
bound to respond to the challenger's case, though would be 
empowered to accept or reject his advice after due considera- 
tion. The degree to which the devil's advocate was able to 
influence substantive judgments in the paper would thus 
ultimately be determined by the NIO. 

3. All papers subjected to this form of challenge 
procedure would bear a notation to that effect somewhere 
(in an introduction, opening statement, footnote, whatever) 

in the paper. In some instances, the burden of the challenger's 
argument (especially if otherwise ignored) could also be 
succinctly stated. For example: 

. 8 
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The principal' conclusions of this estimate 
were subjected to a critical review by the Com- 
munity's "devil's advocate." The contention of 
the advocate that, in the circumstances described 
in the paper, the Arab governments would almost 
certainly sue for peace {because they would be 
principally concerned about their own survival) 
was carefully considered. This argument was > 
however, rejected, in the main because of reasons 
discussed in paragraphs 14-16 of the text. 

4. Occasionally, however, the NIO might wish to 

•h 

reproduce the devil’s advocate's case in extenso T If so, that 
case should be interwoven with the main text. This would 
minimise repetition and confusion and place arguments in a 
proper context. (Most NIEs, etc. , already march in this 
manner, and should continue to do so, though the pros and 
cons of course should not be presented in a way likely to 
obscure the conclusions. ) 


* Indeed, it has been suggested that many papers take the form of 
straight advocacy but provide a separate annex setting forth the 
paper's pros and the devil's advocate's con arguments; or that the 
main text itself should interweave these pros and cons. Adoption 
of the first suggestion--in effect the publication of a devil's annex- 
would, with perhaps very rare exceptions, probably only irritate 
and confuse the consumer. * 


*9 
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5. Whatever the impact of his role on a given 
paper, the devil's advocate should make his case available 
to other elements of the Community (e.g. , current intelligence 
components) so that they, too, could be informed by his 
advocacy. 

25X1 

Chiel^ .Product Review Division 
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